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Longitudinal Perspectives: Expanding the Efficacy of End 
of Course Surveys in Higher Education 

Lawrence Wright 

This qualitative case study examines the use ofend of course surveys in higher education and the value ofstudent percep¬ 
tive data to improve the delivery of online curriculum. The study presents four research questions that examine (1) whether a 
longitudinal analysis of end of course survey data can improve the delivery of curriculum; (2) how student reflected charac¬ 
teristics and attributes of instructors inhibit or contribute to student learning; (3) how student perceptions provide insight into 
the teaching styles of instructors; (4) the efficacy ofqualitative analysis using end ofcourse survey data. Ten end of course 
surveys are used in this study capturing the comments of 114 doctoral students. The survey data are coded revealing themes 
that address perceptions about online instructor characteristics and attributes, perceptions about instructor competence, and 
perceptions about how the instructor delivers curriculum. An analysis of longitudinal data demonstrates that a qualitative re¬ 
search design can be effective in capturing best practices for the self-studied instructor, as well as consideration for expanded 
studies at the institutional level to evaluate the efficacy of program administration, curriculum development, and delivery. 


In higher education, end of course surveys 
(EOCS) are designed to capture both statistical data 
and student comments. Typically, after a course 
ends, results are sent to instructors and identities of 
students who complete the survey are stripped from 
the documents. The survey results are provided to 
instructors with the intent toward self-evaluation 
and improvement and to assess the level of student- 
instructor interaction within the online environment 
(Sher, 2009). The EOCS are also used to evaluate 
instructor performance at the institutional level. 
This qualitative case study provides an analysis of 
EOCS data collected from 10 sections of the same 
doctoral course taught over a two-year period by 
the same instructor. The research provides a thor¬ 
ough, thematic examination of the data collected 
to identify instructor attributes and characteristics 
that encourage student learning, improve instructor 
curriculum delivery, and enhance the student learn¬ 
ing experience. Given the knowledge learned from 
this case study, it is anticipated that the application 
of this knowledge will serve as a positive contribu¬ 


tion to the development of effective online teaching 
strategies and self-monitored, professional develop¬ 
ment, as well as expand upon what can be learned 
from a well-crafted EOCS. 

THE PROBLEM 

EOCS can become powerful self-monitoring 
tools to professionals who are seeking to improve 
their teaching skills. The surveys can also reveal the 
satisfaction and performance of students and their 
ability to connect with their instructors (Miller & 
Redman, 2010). The problem lies in how EOCS data 
are provided to instructors and what instructors are 
expected to do with it. The institution sends survey 
results to instructors shortly after the completion of 
their courses. A longitudinal perspective is not in¬ 
cluded in the EOCS results diminishing the holistic 
view of student perceptions as they pertain to char¬ 
acteristics and attributes that contribute to or inhibit 
student learning. The data does not provide criti¬ 
cal insights into the manner in which the instruc¬ 
tor delivers the curriculum over time, nor does it 
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synthesize student comments that reveal teaching 
strategies that take advantage of strengths or reveal 
weaknesses in the instructor’s teaching pedagogy. 
End of course survey results do not include an aca¬ 
demic program analysis using a similar longitudinal 
perspective, which could provide valuable curricu¬ 
lum development insights especially when multiple 
instructors are evaluated for the same course. 

RATIONALE AND SIGNIFICANCE 

This study employs a longitudinal viewpoint 
that triangulates recurring and cross-cutting themes 
over a two year period. This reflective feedback can¬ 
not come from a quantitative assessment as descrip¬ 
tive language provides clarity about the instructor’s 
pedagogy from the language of student perception. 

Instructor and program administrators be¬ 
come better informed about the kinds of teaching 
attributes that work best in course delivery when 
a long-term perspective is considered. The evalu¬ 
ation of EOCS data rarely extends beyond a se¬ 
mester or term and loses much of its utility as the 
memory of lessons learned are forgotten. The use 
of a longitudinal qualitative research design cap¬ 
tures student perceptions over time, confirming 
commonly perceived learner constructs. The lon¬ 
gitudinal value of this approach improves the use 
of EOCS as feedback instruments, aligns proven 
learning and teaching strategies to the curriculum, 
and contributes to the review of college programs 
as part of the EOCS analysis process. 

Although this study closely examined a single 
instructor and one course taught over a two-year 
period, a similar approach could be used to look at 
all instructors teaching a single course or multiple 
courses over time to capture similar feedback. The 
efficacy of this approach provides another way to 
identify best practices, confirm student perceived 
value of the curriculum, and focus on faculty pro¬ 
fessional development opportunities to correct de¬ 
ficiencies and improve student perceptions. 

THEORETICAL FOUNDATIONS 

Whether online or in traditional classrooms, 
instructors need to understand the psychology of 
student preferred learning styles. This understand¬ 
ing is needed if instructors are to help students link 
course content to the context of their life experienc¬ 
es. This connection helps adult learners put what 
they learn into a familiar context needed to create 


knowledge. Researchers have connected learning 
styles and students into types based on the way stu¬ 
dents like to learn and how students use a range of 
concepts based on the context of their learning en¬ 
vironments (Bach, Haynes, & Smith, 2007). Bach 
et al. (2007) also established the need for students 
to understand their own learning style, thereby as¬ 
sisting them to be more adaptable and to find ways 
to make what they are learning more transferable 
to their working environment. 

In addition to helping students understand how 
they learn, facilitators need to connect life experi¬ 
ences to the curriculum to help learners construct 
new knowledge. Knowles (1988) discussed at length 
the need to design and manage learning activities. 
He identified three key implications for practice that 
orients the learner and depicts how the application of 
learning strategies aligns with adult student learning 
styles. These included “the orientation of adult edu¬ 
cators” with particular emphasis on the logical devel¬ 
opment of curriculum and the need to “attune to the 
existential concerns of the individual and institutions 
they serve and be able to develop learning experi¬ 
ences that wifi be articulated with those concerns” 
(Knowles, 1988, p. 55). The second of three was the 
“organization of the curriculum” (Knowles, 1988, p. 
55). Knowles (1988) emphasized that adult learners 
are oriented toward problem-centered or problem¬ 
solving learning. This focus is on the adult student’s 
ability to apply the material in a practical manner. 
The pragmatism associated with adult learner strate¬ 
gies must be inculcated within the curriculum and the 
teaching pedagogy when presented within an adult 
learning environment. The third point that Knowles 
(1988) related was the “design of the learning expe¬ 
riences” (p. 55). As mentioned, adult learners tend 
toward the problem-oriented learning style which 
implies that “the starting point for every learning ex¬ 
perience is the problem and concerns that the adult 
learner has on their mind at the time the instruction 
is given” (Knowles, 1988, p. 55). This concept is con¬ 
sistent with Merriam, Caffarella, and Baumgartner 
(2007) and the belief that there is a linkage between 
adult interactions within contemporary society and 
the motivation for adult educational opportunities. 
Adults are typically motivated by practical purposes 
either to solve a particular problem or to develop the 
knowledge and competency to increase their abili¬ 
ties to solve problems within their individual and ex¬ 
panded social environments. 
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Knowles, Holton, and Swanson (as cited by 
Stavredes, 2011) contended that adult learners are 
self-directed in that they are apt to diagnose their 
own learning needs and goals, as it relates to eval¬ 
uating their relationship to achieved learning out¬ 
comes. Of particular note was Stavredes’s (2011) 
comments relating to motivation and student per¬ 
sistence in learning; “for learners new to the sub¬ 
ject, with little life experience, or with low motiva¬ 
tion, [these conditions] can lead to a poor learning 
experience and affect their ability to persist” (p. 
14). The literature clearly establishes links between 
adult experience to the subject matter and their mo¬ 
tivation to persist in the learning experience. 

Stavredes (2011) also tied cognitivist, behav- 
iorist, and constructivist learning theories to adult 
learning taxonomies. This included how the cur¬ 
riculum, learning strategies, instructional strate¬ 
gies and curriculum influence how instructors 
focus learners in the building of knowledge. For 
example, Leonard (2002) highlighted the fact that 
constructivism is “learner-centric...in which con¬ 
tent is constructed by the learners in a team-based 
collaborative learning environment rather than that 
of the instructor” (p. 37). Leonard (2002) elaborat¬ 
ed further by suggesting that the instructor is “no 
longer a primary intermediary and single conduit 
of knowledge between the learner and the learning 
experience” (p. 37). Clearly, in the constructivist 
approach, the teacher or instructor is better suited 
as facilitator, which is more in line with non-tradi- 
tional online education. 

There is a compelling alignment between the so¬ 
cial constructivist approach and that of social cognitive 
theory, which describes learning “based on the prem¬ 
ise that neither spontaneous behavior nor reinforce¬ 
ment was necessary for learning to occur” (Snow¬ 
man, 2009, p. 275). Snowman (2009) further stated 
that social cognitive theory “incorporates elements of 
both operant conditioning and information process¬ 
ing, and it emphasizes how behavioral and personal 
factors interact with the social setting in which the be¬ 
havior occurs” (p. 275). Of special interest again is the 
connectedness of problem-oriented education of adult 
learners and the link to practical applications as a key 
motivating force for the adult to construct knowledge. 
Other areas of interest relating to the application of 
social cognitive theory include the concepts of self- 
efficacy, self-regulation, and self-control, all of which 
may have application in this study. 


THE ROLE OF THE INSTRUCTOR 

The role of the instructor in online adult learn¬ 
ing takes on a facilitative, pedagogical posture 
within the online classroom. The instructor’s role 
is focused on guiding the student through modular 
curriculum. Key to curricular design is the need 
to motivate student engagement in online learning 
activities. Horton (2012) described the kinds of in¬ 
structional strategies online instructors can use to 
engage students in online learning activities. These 
patterns of interaction include a tutoring one-to-one 
or a one-to-many relationship where the instructor 
provides common information to all students and 
a question and answer didactic learning strategy 
similar to that used in traditional classrooms. Ryan, 
Cooper, and Tauer (2008) explained that appropri¬ 
ate feedback is key to ensure that students are ac¬ 
tively learning the curriculum. 

For instructors to give appropriate feedback, 
they need to learn how to decode student behavior 
and messaging to assess whether student learning 
is occurring. In the traditional classroom, this kind 
of communication model is much more robust be¬ 
cause the instructor can use both verbal and non¬ 
verbal cues to formatively assess student learning. 
Horton (2012) also suggested that instructors in on¬ 
line environments use a didactic pattern of ques¬ 
tions and answers. This would be more than asking 
the student a question and then leaving the student 
to provide an answer without engaging the student 
in a discussion. Horton (2012) discussed the need 
for active engagement between the instructor and 
the student to provoke student learning. The key to 
Horton’s (2012) comments is the active engagement 
of the instructor to facilitate active learning whether 
it is individual or collaborative. The idea is to pro¬ 
vide iterative feedback to keep adult learners en¬ 
gaged, especially when that interaction ties student 
interests to problem-solving and problem-oriented 
discussion that is relevant to the student’s needs. 

According to Clark and Mayer (2011), instruc¬ 
tors need to be psychologically visible to the stu¬ 
dent. Clark and Mayer (2011) suggested, “There 
is preliminary evidence that using the visible au¬ 
thor style can promote deeper engagement in some 
learners” (p. 200). Apparently, this human connec¬ 
tion provides a source of motivation to students 
where “social presence encourages the learner 
to engage in deeper cognitive processing during 
learning, leading to a better learning outcome” 
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(Clark & Mayer, 2011, p. 200). Clark and Mayer’s 
(2011) emphasis on a psychologically visible pres¬ 
ence in the online classroom underscores the need 
for the instructor to actively motivate the student 
to learn. 

Instructors benefit when introduced to the At¬ 
tention Relevance Confidence Satisfaction (ARCS) 
Model (Keller, 2009), which can guide faculty to 
increase student motivation for learning by incor¬ 
porating motivational approaches into their lesson 
planning and presentation. The model includes four 
steps or categories that can be used to stimulate 
knowledge creation. These include getting students’ 
attention, or “capturing the interest of learners and 
stimulating their curiosity to learn,” making the 
curriculum content relevant, encouraging students 
as they progress through the curriculum with “con¬ 
fidence,” and “reinforcing their accomplishments” 
by helping them feel satisfaction in their learning 
experience (Keller, 2009, p. 44). For adult learners, 
keeping the primary focus on why the learner is 
attending the class and what their end objectives 
are provide the leverage for instructors to apply the 
ARCS Model. This model fits nicely with Gagne’s 
(Gagne, Wager, Golas, & Keller, 2005) nine events 
of instruction. These nine events focus on (1) gain¬ 
ing the attention of the learner; (2) informing the 
learner of the learning objective; (3) stimulating 
recall of prior learning (critical to tying prior ex¬ 
perience to the lesson plan of an adult learner); (4) 
presenting the curriculum in a stimulating manner; 
(5) providing learners guidance; (6) eliciting and 
providing feedback;(7) assessing student perfor¬ 
mance; (8) looking for ways to enhance the reten¬ 
tion of knowledge learned; and (9) appealing to the 
interests of the learner. 

END OF COURSE SURVEYS 

End of course surveys (EOCS) have been used 
within higher education to evaluate the quality of 
traditional and online instruction. Dittmar and Mc¬ 
Cracken (2012) recently studied the use of what 
they call the META Model. This model uses a strat¬ 
egy that captures, as part of a continuous quality 
improvement model in online teaching, specific 
elements that can concentrate on professional en¬ 
gagement and development. It uses an ongoing as¬ 
sessment to measure peer individual faculty or self, 
along with student evaluations like EOCS to cap¬ 
ture exemplary practices. It is worth noting that the 


aims of this model are to develop a cadre of high- 
performing faculty capable of “fostering rigorous 
learning” and “programs [that] strengthen both ef¬ 
fectiveness and productivity” (Dittmar & McCrack¬ 
en, 2012, p. 164). The model centered its continuous 
improvement metrics on faculty, in areas such as 
student mentoring and technology, by integrating 
emerging technologies such as Web 2.0 tools and 
platforms, engagement through training and per¬ 
formance improvement, and assessment, which in¬ 
cludes the use of an exemplary teaching rubric. 

Drennan, Kennedy, and Pisarki (2005) linked 
the concept of flexible online learning to the idea 
that “students should be viewed as active partici¬ 
pants in the learning process and that deeper ap¬ 
proaches to learning should be encouraged” (p. 
332). From their research, they synthesized two 
basic attributes that influence student satisfaction 
affecting the student’s locus of control and their 
perceived usefulness of the course content and cur¬ 
riculum presentation. These perceptive attributes or 
factors focused on the student’s use of technology 
within the learning environment and “autonomous 
and innovative learning styles” (Drennan, Ken¬ 
nedy, & Pisarki, 2005, p. 331). Student satisfaction 
increased as student perception of the usefulness of 
curriculum content improved. The use of technol¬ 
ogy by instructors and students also facilitated the 
positive perceptions of curriculum usefulness and 
satisfaction. The implications from their study di¬ 
rectly relate to the facilitative role, or learner-cen¬ 
tered role, faculty enact in teaching online courses. 

Student satisfaction perceptions were also 
studied by Getzlaf, Perry, Toffner, Lamarche, and 
Edwards (2009) who claim that student percep¬ 
tions are tied to “effective instructor feedback” (p. 
1). Their qualitative study examined student- in¬ 
structor online feedback and identified five major 
themes that tie into the instructor’s ability to capi¬ 
talize on learner-centered, flexible learning. These 
major themes are “student involvement/individual¬ 
ization, gentle guidance, being positively construc¬ 
tive, timeliness and future orientation” (Getzlaf et 
al., 2009, p. 16). A surprising finding in the Getzlaf 
et al. (2009) study was the student belief that ef¬ 
fective instructor feedback was a mutual activity 
performed between instructor and student, and it 
was this mutual connection that was instrumental 
in helping students find relevance with the curricu¬ 
lum. A similar finding by Jones (2012) who con- 
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ducted a study that looked closely at the validity of 
EOCS data found that “stimulation of learning had 
the most effect on perceptions of teaching effec¬ 
tiveness” by student online learners (p. 49). 

PURPOSE 

The purpose of this study is to evaluate the ef¬ 
ficacy and use of a qualitative research design to 
capture themes that illuminate means of improving 
the delivery of course curriculum from longitudinal 
EOCS data. Along that line, the research methodol¬ 
ogy addresses four cross-cutting research questions 
that focus on the longitudinal potential of thematic 
qualitative research design. The remaining research 
questions draw from the data information that re¬ 
veals the interrelationship of students with their in¬ 
structor, the characteristics students perceive exists 
in their instructor that may contribute or inhibit the 
student learning experience, and student percep¬ 
tions on instructor delivery of the curriculum. The 
collection and analysis of this information confirm 
the efficacy of longitudinal EOCS data analysis and 
qualitative research design. 

RESEARCH QUESTIONS 

The overarching research emphasis for this 
case study addresses the efficacy and use of this 
qualitative approach and research design in captur¬ 
ing themes that can improve the delivery of course 
curriculum. The EOCS used in this study contained 
one narrative question that became the source for 
analysis by these four research questions: 

• Rl: Can a longitudinal and more detailed 
analysis of student perceptive comments 
found within EOCS provide more insight 
into student learning than looking at individ¬ 
ual EOCS comments? 

• R2: What are the student-reflected charac¬ 
teristics and attributes of the instructor that 
may be contributing to or inhibiting student 
learning? 

• R3: What are student perceptions relating to 
the manner in which the instructor delivers 
the curriculum? 

• R4: What can be synthesized from student 
comments that have the potential for improv¬ 
ing learning strategies that may take advan¬ 
tage of the strengths or address weaknesses 
discovered through qualitative analysis of 
student perceptive data? 


METHODS 

The methodology and design of this study fo¬ 
cused on answering the research questions using a 
robust, qualitative research case study design. The 
coding process applied axial coding logic (Strauss 
& Corbin, 1998). This involves coding the actual 
words used by students in their EOCS comments 
and then looking for conceptualizations that help 
answer the research questions. In essence, the 
cross-cutting themes represented in the research 
questions were categorized and subcategorized to 
illuminate the phenomenon. Miles and Huberman 
(1994) stressed the importance of coding by “noting 
reflections,” identifying relationships, patterns and 
themes, and isolating those themes to a small set 
of generalizations that reveal “commonalities and 
differences” (p. 9). This approach struck a com¬ 
mon qualitative design theme acknowledged by 
Richards and Morse (2007). Richards and Morse 
(2007) suggest six steps or “moves” that frame the 
research design used in a qualitative study: 

(1) Coding the data as data records are created; 
(2) recording reflections and insights; (3) sorting and 
sifting through the data to identify similar phrases, 
relationships, patterns, themes, distinguishing fea¬ 
tures, and common sequences; (4) seeking patterns 
or processes, commonalities and differences and 
extracting them for subsequent analysis; (5) gradu¬ 
ally elaborating a small set of generalizations that 
cover the consistencies discerned in the database; 
and, (6) confronting these generalizations with a 
formalized body of knowledge in the form of con¬ 
structs or theories, (p. 47) 

DESIGN 

The research design utilized the following steps: 

1. EOCS PDFs were imported into a qualitative 
software tool called NVivo (QSR Internation¬ 
al, 2012). NVivo software is a popular and ro¬ 
bust qualitative research tool used to analyze 
and code text and multimedia sources. 

2. Each research question was coded separately 
and stored in a parent node. Thematic sub¬ 
categories were added in vivo. 

3. A second level of coding occurred using 
common themes found in each of the parent 
nodes. These themes were coded and stored 
in newly created child nodes to further de¬ 
lineate more structured cross-cutting themes 
and relationships. 
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4. Child nodes were analyzed using queries 
or reports. Queries created in NVivo were 
linked to their respective nodes allowing 
the researcher to drill down deeper into 
the analysis. Additional nodes were created 
from queries capturing cross-cutting themes 
and data relationships. For example, a Word 
Frequency Query was used to illustrate com¬ 
mon student perceptions. Other illustrations 
were created to provide clarifications on stu¬ 
dent perceptions as they related to thematic 
subcategories coded in child nodes. 

5. The results were compared again to the the¬ 
oretical literature. 

6. A narrative was written to reflect what the 
researcher learned from the analysis, leading 
to answers for each of the research questions 
and opportunities for future research. 

PROCEDURE 

Data collection began in 2010 and ended in 
2012. In Table 1, the month and year are indicated 
along with the number of participants who com¬ 
pleted the survey. The EOCS instrument used in 
the study has 14 questions. Of the 14 questions only 
one question was open ended, allowing students to 
provide an unstructured narrative response. Thir¬ 
teen questions utilized a five-point Likert-scale and 
were not used in the study. The open-ended ques¬ 
tion asked students the “reason that you would or 
would not recommend this instructor?” The narra¬ 
tive responses were imported into NVivo for cod¬ 
ing. Only the open-ended question was used for 
this study. 

Table 1: Participants 

End of Course Survey Participants 

E0CS_N0V_12 10 

E0CS_SEP_12 9 

E0CS_SEP_12 6 

E0CS_AUG_12 3 

E0CSJUL12 11 

E0CS_JUN_12 14 

E0CS_APR_12 17 

E0CS_FEBJ2 18 

E0CS_N0V_11 14 

E0CS_SU1_10 12 

Total Surveys: 10 Total Participants: 114 


PARTICIPANTS 

The participants were “purposely selected” (Cre- 
swell, 2003, p. 185). This means that the researcher 
had selected participants that “can best help the re¬ 
searcher understand the problem and the research 
question” (Creswell, 2003, p. 185). The majority of 
participants were employed in education adminis¬ 
tration at the primary and secondary levels. There 
is a smaller subset of students who were healthcare 
professionals, and then a yet smaller group in the 
doctoral management leadership track. 

Generally, these students were mid- to upper- 
level career adults and enjoyed considerable experi¬ 
ence in online education with many completing on¬ 
line master’s degrees. All students were given the 
option to self-select participation. Not all students 
attending these 10 courses completed the surveys. 
The average number enrolled per class was 20 with 
the potential for 200 participants. Of the 200 poten¬ 
tial participants, 114 completed the EOCS. We can 
assume that those who completed the survey were 
motivated to do so. The numbers of participants 
by survey or course is listed in Table 1. Individual 
identities were removed by the university providing 
the data for this study. The university has also given 
permission to use the data on the condition it not 
be named, further protecting the participants. The 
researcher cannot link the data to the participant. 

CODING 

Embedded in the Richards and Morse (2007) 
steps is the undergirding of axial coding, which 
includes the categorization and conceptualization 
of data. The use of NVivo 10 (QSR International, 
2012), a qualitative analysis software tool, was used 
to create coding nodes that reflect researcher ob¬ 
servations and insights that were categorized and 
subcategorized for common or similar phrases, se¬ 
quences, patterns, and themes. Narrative responses 
were coded seeking to conceptualize themes from 
the initial iteration to a second iteration, framing 
the results from general to specific themes. After 
coding the results, the researcher revisited the lit¬ 
erature to further triangulate related concepts that 
confirmed the coded analysis. 

DATA ANALYSIS 

Nodes were created reflecting the perspectives 
of student responses to the open-ended EOCS ques¬ 
tion during the coding process. The initial coding 
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focused on answering the four research questions. 
Coded responses were collected into parent nodes. 
A second round of coding focused on subcatego¬ 
ries, which naturally emerged during the coding 
analysis. These subcategories were coded as sepa¬ 
rate child nodes revealing the word essence of what 
students were describing. Cross-cutting themes 
emerged and are reflected more pronouncedly in 
research questions R2 and R3. The longitudinal in¬ 
sights reflected in R1 and the student improvement 
insights reflected in R4 clearly underscore the ef¬ 
ficacious relationship of the data analysis. Figure 1 
depicts the data analysis process. Figure 2 provides 
a graphic representation of the node hierarchy. 

There were five subcategories that emerged 
from the second research question node during the 
analysis. These included a subcategory called “car¬ 
ing,” which was a recurring theme from students. 
The second category included negative attitudes or 
characteristics, which seemed to emerge as a sepa¬ 
rate category. The third was the positive attitudes 
or characteristics students expressed about the in¬ 
structor. The last two subcategories related to stu¬ 
dent feelings about the instructor’s competency as 
it relates to knowledge of the curriculum and spe¬ 
cific descriptions that highlight their recommenda¬ 
tions or willingness to recommend the instructor 
to their peers. 

There were six subcategories that emerged from 
the data relating to student perceptions about the 
instructor’s ability to deliver the curriculum. These 
focused on the way the instructor communicated to 
students, how the instructor facilitated online dis¬ 
cussions, the nature of grading feedback, perceived 
learning objectives, recognizable teaching styles, 
and how students felt about the feedback received 
from graded assignments. Student comments also 
indicated whether the instructor’s teaching style 
lead to specific or perceived learning outcomes. In 
addition, the last two subcategories identified from 
student comments indicate a distinct or recogniz¬ 
able instructor teaching style and perceptions on 
how the instructor approached and managed the 
online classroom. 


Figure 1: Research analysis 

1) Data imported into NVIVO 

2) Initital coding creating parent nodes 
using research questions for structure 

3) Second level coding creating child 
nodes or sub-categories 

4) Analysis 

5) Revisiting the literature 

6) Writing the research narrative 

Figure 2: Nodes 

Nodes 

'V Name 

0 Q R1 - Longitudinal Insights 

Q Student Attitudes about the University 
g O R2 - Student Perceptions of Instructor Characteristics and Attribute; 
Q Caring 

fO Negative Attitudes and Characteristics 
o Positive Attitudes and Characteristics 
o Relating to Knowledge 
Q Student Descriptions of Recommendation 

B O R3- Student Perceptions Relating to Instructor Delivery 

Q Communication 
Q Discussion Facilitation 
Q Grading Feedback 
o Perceived Learning Outcomes 
Q Recognizable Teaching Styles 
Q Student perceptions on classroom management 

Q R4 - Student Improvement Insights 


SECOND LEVEL CODING 

By referring again to Figure 2, the relationship 
between the parent and child nodes becomes ap¬ 
parent. Child nodes were created in vivo, meaning 
they were created on the fly as the subcategories 
were revealed to the researcher. Another example 
of the coding relationships existent between the 
parent, child, and sources is illustrated in Figure 3. 
Note the relationships between the child nodes to 
the R2 research question and the EOCS PDFs. The 
data analysis relationships are fully aligned, main¬ 
taining validity and reliability in design structure 
and analysis. 
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Figure 3 R2: Node relationships 


Table 2: Word Frequency List 



ANALYSIS PROCESS 

Child nodes were analyzed with the help of 
NVivo queries. These included the child nodes 
for each of the parent nodes illustrated in Figure 
2. The analysis started when the researcher began 
coding and categorizing the data into nodes. Once 
the coding was complete, a series of NVivo queries 
were generated. Through some trial and error, the 
researcher found queries relating to the most fre¬ 
quent descriptive words to be the most helpful. For 
example, the Word Frequencies Query in Figure 5 
are linked to their respective nodes allowing the re¬ 
searcher to drill down into the data to better under¬ 
stand the context associated with the usage of each 
particular word. This is illustrated with the use of 
the word “feedback” as one of the most frequent¬ 
ly used words by students in the EOCS. Figure 4 
describes 15 of the most frequently used words in 
student comments. The figure also provides a spa- 
cial view of the most commonly used words. The 
Word Frequency List is also illustrated in Table 
2. The key to reviewing the Word Frequency List 
is to graphically picture the spacial relationships 
of key words illustrated in Figure 4 (below). The 
Tree Map illustrates graphically and spatially the 
use of 15 words mentioned in Figure 4. A similar 
analysis was also provided with a cluster analysis 
in Figure 5. The researcher was able to drill down 
to further examine student perceptions in context 
from all 10 of the EOCS source documents. Word 
lists were linked to their coded narrative examples. 
The weighted percentage on the use of descriptive 
words can be seen in Table 2 (below). 


Weighted 


Word 

Length 

Count 

Percentage 

feedback 

8 

197 

3.17 

helpful 

7 

82 

1.32 

questions 

9 

71 

1.14 

assignments 

11 

70 

1.13 

great 

5 

70 

1.13 

time 

4 

68 

1.10 

engaging 

8 

66 

1.06 

gave 

4 

65 

1.05 

learning 

8 

60 

0.97 

discussion 

10 

54 

0.87 

always 

6 

52 

0.84 

timely 

6 

50 

0.81 

recommend 

9 

49 

0.79 

engaged 

7 

41 

0.66 

discussions 

11 

39 

0.63 


Figure 4: Tree map 15 (wordfrequency spacial view) 
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Figure 5: Word frequency query 
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REVISITING AND COMPARING CODED RESULTS WITH 
THE LITERATURE 

Following the coded analysis, the researcher 
again examined the literature. Many of the sources 
examined were downloaded as PDF source docu¬ 
ments and were loaded into the NVivo database. 
Simple word comparisons of the literature of de¬ 
scriptive words used by students gave the research¬ 
er the opportunity to pinpoint the significance from 
prior research as articulated in the literature to the 
coded analysis. 

WRITING THE NARRATIVE 

The narrative was written after an analysis of 
the coding and a second look at the literature was 
completed. The author was careful not to allow the 
initial literature review to bias the coding. The sec¬ 
ond review of the literature allowed for a compari¬ 
son of coded analysis with the body of knowledge 
represented in the literature. 

RESULTS 

The overarching research question was Rl, 
which addressed the efficacy of including a longi¬ 
tudinal perspective when providing EOCS data to 
instructors. Questions R2 through R4 underscore 
what could be learned should a longitudinal per¬ 
spective frame the analysis and are included when 
EOCS results are distributed to instructors follow¬ 
ing the completion of their courses. 

Rl: LONGITUDINAL INSIGHTS 

The first and overarching research question 
asked: can a longitudinal and more detailed analy¬ 
sis of EOCS student perceptive comments provide 
more insight into student learning than looking at 
individual EOCS comments? It was clear from the 
analysis that a longitudinal view of EOCS data over 
the two-year period provided an outlet allowing for 
suggestions and some student venting of steam on 
issues that prevented or enhanced student learn¬ 
ing experiences at the administrative level. Student 
insights underscore the diversity of comments and 
the importance of those comments communicated 
in the survey and coded in the Rl: Longitudinal 
Insights node. Coded comments gave examples of 
basic attitudes about the university. Many also il¬ 
lustrated a mental image of the student’s position 
or relative relationship to the university at the time 
the survey was taken. 


R2: STUDENT PERCEPTIONS OF INSTRUCTOR 
CHARACTERISTICS 

The second research question asked: what are 
the student-reflected characteristics and attributes 
of the instructor that may be contributing to or 
inhibiting student learning? Five sub-categories 
emerged from the coded data that related to the 
instructor’s ability to contribute or inhibit student 
learning. These subcategories include the percep¬ 
tion of caring, profound negative expressions of 
dissatisfaction, positive attributes and characteris¬ 
tics, comments relating to the instructor’s knowl¬ 
edge and competence, and student expressions of 
recommendation. Each of these is addressed sepa¬ 
rately below. 

Caring: The idea that the instructor cared gen¬ 
uinely for his students was reiterated in student 
comments. The concept of caring was shared as a 
caring attitude toward how students were treated 
in the class and the belief by students that the in¬ 
structor was genuinely committed to their success. 
These perceptions certainly open opportunities 
for further research studying the emotional con¬ 
nections that may lead to student performance and 
engagement, as well as the impact on student reten¬ 
tion. 

Negative attitudes and characteristics: During 
coding there were only a few students who ex¬ 
pressed dissatisfaction. In order to not allow bias 
to filter these comments, they were included for the 
intrinsic value they had. These student perceptions 
were not consistent with the pervasive feedback ex¬ 
pressed by the majority of students. Nevertheless, 
they underscore perceptions that are important to 
articulate here. Most of the feedback related to is¬ 
sues surrounding communication, feedback, lack 
of clarity pertaining to assignment instructions, 
and a lack of consistency relating to grading. 

Positive attitudes and characteristics: Positive 
attitudes and characteristics were more pervasive 
throughout the EOCS data. It was common to read 
comments that illustrated a positive relationship 
with the instructor communicating expressions 
of engagement, availability, helpfulness, encour¬ 
agement and understanding, approachability, and 
clarity in communicating expectations. Also noted 
was the perception that the instructor has a sense of 
humor in his communications or associations with 
students. These comments seemed to illustrate a 
personal connection students had with the instruc- 
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tor. For example, one student stated, “In this imper¬ 
sonal, online format, he made it easy for me to feel 
as though he knew me, my abilities, and pushed me 
to be better. I was concerned about the online for¬ 
mat and, because of Dr. [name], am much less so.” 

Relating to knowledge/perception of compe¬ 
tence: Throughout the coding process there were 
comments made by students that related to their 
perception of the competence of the instructor. 
This perception appeared to illustrate their view 
about the instructor’s ability to contribute to or in¬ 
hibit student learning. Coded examples appeared to 
highlight that the instructor was perceived as being 
knowledgeable, qualified, accessible, knowledge¬ 
able pertaining to the course curriculum, fluent in 
the curriculum, authentic, reasonable, and had the 
ability to help students achieve their goals. Stu¬ 
dents also equated “knowledgeable” to the ability 
of the instructor to teach curriculum in a way that 
made it “understandable to life situations.” As one 
looks at some of the coded examples, it is evident 
that student perceptions were not unanimous. In at 
least two cases, students did not understand what 
the instructor was trying to communicate, which 
may point to an inhibitor. 

Student expressions of recommendation: It be¬ 
came apparent that student expressions of recom¬ 
mendation mirrored their perceptions of whether 
the instructor was contributing to or inhibiting 
their academic experience. Comments usually il¬ 
lustrated a semantic descriptor that expressed the 
reason for their recommendation. In nearly all 
cases, the comments articulated characteristics and 
attributes. Some of these included recommenda¬ 
tions relating to the instructor’s leadership ability. 
For example, a student statement claiming that the 
instructor had “charisma.” Other comments of rec¬ 
ommendation reflected sentiments of knowledge 
and competence, precision, ability to support, offer 
of guidance, ability to teach or instruct, ability to 
provide useful feedback, and ability to relate the 
curriculum to life’s experiences. 

R3: STUDENT PERCEPTIONS RELATING TO 
INSTRUCTOR DELIVERY 

The third research question asked: what are 
student perceptions relating to the manner in which 
the instructor delivers the curriculum? The answers 
learned from this research question were quite re¬ 
vealing. During the coding process, six subcatego¬ 


ries surfaced. These include communication, dis¬ 
cussion facilitation, grading feedback, perceived 
learning objectives, recognizable teaching styles, 
and student beliefs that the instructor was assisting 
them in learning the curriculum. Specific teaching 
styles were identified in student perceptions about 
how the instructor managed the online learning en¬ 
vironment. 

Communication: This subcategory addressed 
cross-cutting themes that also overlapped with the 
other subcategories. Nevertheless, there appeared 
to be a series of characteristics and attributes that 
related to the overall student perception describ¬ 
ing how the instructor communicated. There are 
a number of adjectives used to describe the com¬ 
munication model used by the instructor. Students 
typically saw criticism as being constructive. The 
instructor was perceived as being engaging and 
providing feedback in helpful ways. The instruc¬ 
tor was perceived as one who asked probing ques¬ 
tions. Communication was perceived to be helpful, 
informative, and task centered. Students perceived 
communication to be relevant. Another interest¬ 
ing finding was how the students perceived their 
relationship or status to the instructor. Comments 
like “he actually treated all the students as if they 
were adults” were surprising, and it certainly begs 
for more investigation of faculty interactions as a 
whole. Students frequently mentioned accessibility 
and the willingness of the instructor to call students 
on the telephone as indicative. Students also men¬ 
tioned that communication was clear and balanced. 
It was surprising to see students expressing opin¬ 
ions about how they felt about their communication 
experiences with the instructor. Comments such as 
“I love the personal contact” and “I enjoyed read¬ 
ing his feedback to other students’ posts” were re¬ 
flective of the communication approach perceived 
by students of their instructor. 

Discussion facilitation: Overall, students de¬ 
scribed the instructor as engaged with meaningful 
feedback. This was the most frequent description. 
Other descriptions included the use of words or 
phrases such as probing questions, relevant, chal¬ 
lenged and encouraging, and helpful and engaged. 
In addition to being helpful, students also felt that 
discussion posts included humor. Students de¬ 
scribed the discussion facilitation techniques used 
by the instructor as being interactive and inclusive. 

Students also saw the instructor’s facilitation 
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approach as “forcing the student to think outside 
the box.” It was clear from responses that most of 
the students were responding successfully to com¬ 
munication and motivational approaches used by 
the instructor. 

Grading feedback: The need for grading feed¬ 
back emerged as a subcategory during coding. The 
course required five research papers, all of which 
provided an iterative learning experience as stu¬ 
dents applied what they learned toward the next as¬ 
signment. The fact that grading feedback emerged 
as an evaluative component in their comments of 
instructor performance was not surprising. Com¬ 
ments were descriptive of the type and value of the 
feedback, as well as its timeliness and relevance. 
Students also indicated the importance of individu¬ 
alized feedback and feedback that helped them im¬ 
prove their writing skills. One student expressed 
frustration that the feedback, which was charac¬ 
terized as good, was not always clear. However, 
largely, the comments from students communi¬ 
cated approval of the instructor’s grading feedback 
approach. 

Perceived learning objectives: Another inter¬ 
esting discovery was the perception that interac¬ 
tion with the instructor helped achieve the learn¬ 
ing objectives of the course. The idea here is that 
the instructor instilled a belief that what students 
were studying helped them achieve their learning 
outcomes and program goals. Frequently, students 
linked engagement activities to their ability to con¬ 
tinue through the doctoral program or to pursue 
other scholarly activities. 

Recognizable teaching styles: Throughout the 
coding process there were comments made by stu¬ 
dents that helped to create an emerging view of the 
instructor’s teaching style. Key words provided this 
window for observation and underscore the impor¬ 
tance and analysis of EOCS qualitative data. For 
example, students characterized the teaching style 
of the instructor as accessible, engaging, assisting 
students as needed, willing to help, and returning 
emails and phone calls in a timely manner. Fre¬ 
quently, students referred to the instructor as en¬ 
gaging, encouraging, and understanding. Students 
seemed to value those characteristics and attached 
to feedback the idea of timeliness. A few students 
underscored this point when they indicated that in 
their situation feedback was not timely or occurred 
after the fact. It was not enough for them to receive 


good feedback if it did not come in a timely man¬ 
ner. The issue of expertise and knowledge of the 
curriculum became another important point. The 
use of probing questions and online discussion en¬ 
gagement was mentioned. Students wanted to be 
treated like adults. The personal and emotional 
connections the instructor made with students were 
obvious in student comments. Students liked to be 
challenged or “pushed” by the instructor, but they 
also acknowledged that the instructor was not over¬ 
bearing. Students also perceived that the instructor 
was willing to help them resolve their concerns by 
taking the necessary time to get them the answers 
they needed. 

Student perceptions of classroom management: 
Another cross-cutting theme was how the online 
classroom was managed. Comments discussing 
instructor actions in communications, clarity of 
instructions, clarity on assignment requirements, 
feedback, grading feedback, timeliness of feed¬ 
back, the feel of the classroom atmosphere, the 
emotional ties or relationships of students to the 
instructor, and the flexibility of the instructor to 
accept assignments past due dates. The instruc¬ 
tor’s sense of humor, engagement, and interactivity 
also seemed to contribute to students’ perceptions 
of the instructor being present and engaged within 
the online classroom. 

R4: Student Improvement Insights: The last 
and fourth research question asked: what can be 
synthesized from student comments that have the 
potential for improving learning strategies and that 
may take advantage of the strengths or help correct 
weaknesses discovered through qualitative analysis 
of student perceptive data? Like Rl, this research 
question captured student improvement suggestions 
that focused on their scholarly pursuits, the difficul¬ 
ties associated with the use of the learning manage¬ 
ment system, the need for more education-related 
literature in the online library, clarity of assign¬ 
ments, recommendations for a more content- ver¬ 
sus format-related grading rubric, and suggestions 
on how to better use the questions for the teacher 
public discussion forum. Some students were pre¬ 
occupied with the way e-books were made avail¬ 
able. The number and frequency of discussion posts 
seemed to be a burning topic for some students. The 
kinds of comments that were coded in this category 
provided insights into how students are reacting to 
the learning management system, the pedagogy as- 
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sociated with the presentation of the curriculum, 
and the attitudes associated with the way the doc¬ 
toral program is organized and administered. Many 
of the comments were cross cutting. 

DISCUSSION 

The most aligned study from the literature that 
seemed to mirror this case study was that of Get- 
zlaf et al. (2009). Getzlaf et al. (2009) looked deep¬ 
ly into online graduate students’ perceptions about 
what constituted effective instructor feedback. The 
study found five recurring themes that seemed to 
resonate with the coding completed in this study. 
The themes pointed to instructor efforts to engage 
students in the curriculum. There seemed to be an 
interaction between the student and the instructor 
that gave the impression of individualization and 
emotional ties and concerns for the success of the 
student. In this study, students expressed the notion 
that they were receptive to feedback when the in¬ 
structor was not overbearing. Getzlaf et al. (2009) 
indicated that feedback was tied to “gentle guid¬ 
ance.” 

The notion of timeliness associated with feed¬ 
back was also evident in the Getzlaf et al. (2009) 
study, and this finding was validated in this study 
as students expressed appreciation when feedback 
was given quickly and when, in a few cases, feed¬ 
back was not given in time for student use while 
preparing for an assignment. When we look close¬ 
ly at the Getzlaf et al. (2009) study, we can also 
see a correlation between the increase in student 
frequency in the use of the word “feedback” and 
the emotional expressions related to the student-in¬ 
structor interaction associated with feedback. This 
ties nicely with Getzlaf et al.’s (2009) discovery that 
feedback is perceived as a mutual relationship, and, 
from this study, how feedback plays a key role in 
how students perceive the interaction between their 
style of learning and the instructor’s teaching style. 


CONCLUSIONS 

There are a number of lessons or implications 
to improving online learning that can be ascer¬ 
tained from this study. One of the most powerful 
implications relates to the efficacy of the qualita¬ 
tive research model to guide instructors through 
a thorough and meaningful review of their EOCS 
data. The model provided another approach to ana¬ 


lyze the rich narrative captured from student per¬ 
ceptions through the use of open-ended questions. 
There should be consideration and further study 
given to modifying EOCSs to include additional 
semi-structured, open-ended survey questions that 
can be analyzed using a qualitative research de¬ 
sign. These questions should address the relevance 
of coursework content within a program of study, 
usefulness of content to a student’s profession, and 
student perceptions on how the instructor delivers 
the curriculum. The use of meta-studies conducted 
at the department level, or by program or course, 
could yield valuable pedagogical best practices, as 
well as types of practices that should be avoided. 

Although this study focused on one instructor 
and his relationship to a particular course taught 
over time, the use of this qualitative research de¬ 
sign should be considered in examining one course 
taught by all instructors within a term or semester, 
as well as over a predetermined period. The out¬ 
comes may yield a more generalized result captur¬ 
ing common applications to all faculty engaged in 
teaching the same course. The methodology may 
also be used to examine all courses taught within 
a department, providing a departmental view that 
can be translated into performance indicators and 
developmental curriculum. 

Coded responses revealed student perceptions 
about the character and attributes of the instruc¬ 
tor and student feelings expressed in likes and dis¬ 
likes about the instructor’s competence and teach¬ 
ing style. The instructor’s ability, as perceived by 
students, to deliver the curriculum was the most 
revealing. Again, there were cross-cutting themes 
revealing student perceptions on the mechanics of 
how the curriculum was delivered. This included 
the way the instructor communicated, the student’s 
perception of instructor competence, the positive 
and negative perceptions occurring in discussion 
facilitation, insights into the magnitude in which 
students developed personal relationships with the 
instructor, the perceived value of receiving grading 
feedback, and the employment of classroom man¬ 
agement techniques. The research also revealed 
when students recognized that they were learning 
and the characteristics associated with the instruc¬ 
tor’s teaching style that was most influential in the 
enhancement of instructor curriculum delivery. 

Another surprise was the richness and depth 
associated with the data gathered from one nar- 
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rative question on the EOCS. Clearly, the use of 
EOCS data can provide valuable feedback when 
analysis includes a longitudinal viewpoint. Institu¬ 
tions who use EOCS data may wish to expand the 
use of open-ended questions to target qualitative 
data of concern. 

Several limitations surfaced during the study. 
The study did not examine the motivation that lead 
students to complete their EOCS at the end of the 
course. Clearly, more participation would lead to 
more data from which to triangulate an analysis. 
However, there was enough participation from the 
dataset used to get meaningful answers to the re¬ 
search questions. 

Another limitation in the study was the poten¬ 
tial for researcher bias during coding and analysis. 
Researcher bias can enter into the coding and anal¬ 
ysis process by instructor-researchers when data is 
encountered that reflects poorly on the instructor. 
If coding was done by someone other than the in¬ 
structor, the potential for bias will be minimized. 
Another limitation centered on the student popula¬ 
tion, which exclusively consisted of doctoral stu¬ 
dents. The efficacy of this methodology will need 
to be replicated studying both undergraduate and 
graduate students. Can the results from this study 
be extrapolated to other populations? Possibly. 
However, the purpose of the study centered on the 
efficacy and use of the EOCS results for self-re- 
flection by a doctoral instructor. Can this research 
design help online instructors gain a better under¬ 
standing of how their teaching styles are serving 
students? Certainly. 

Inferences leading to changes in how EOCS 
are promulgated and used by the university should 
be considered for future study. Additional research 
is needed to confirm the findings of this study and 
the use of qualitative research methods to evaluate 
EOCS data from a longitudinal viewpoint. 
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